Driving assistant system

ABSTRACT

Driving assistant systems and computer-implemented methods for improving landmark-based route guidance are provided. The computer-implemented method includes receiving, by a processor, image data including one or more landmarks. The computer-implemented method further includes identifying, by the processor, a candidate landmark within the image data. The computer-implemented method further includes presenting, by the processor, the candidate landmark to a user. The computer-implemented method further includes, in response to the user accepting the candidate landmark, storing, by the processor, the candidate landmark and a location of the candidate landmark.

DOMESTIC AND/OR FOREIGN PRIORITY

This application is a divisional of U.S. application Ser. No.15/350,381, titled “DRIVING ASSISTANT SYSTEM” filed Nov. 14, 2016, theentire contents of which are incorporated herein by reference.

BACKGROUND

The present invention relates in general to navigation systems, mapping,and route guidance. More specifically, the present invention relates toa driving assistant system and computer-implemented method for improvedlandmark-based route guidance functionality.

Automotive navigation systems (also known as route guidance systems) areoften satellite-based navigation systems designed for use in private andcommercial vehicles. Most systems use a combination of GlobalPositioning System (GPS) data and digital map-matching to calculate avariety of routes, such as the shortest route to a specifieddestination. The systems conventionally present a map overview andturn-by-turn instructions to drivers using a combination of auditory andvisual information. Navigation systems typically equipped in a vehiclegenerally guide a driver by indicating a distance and an intersectionname at which the driver is supposed to make a turn, e.g., “turn rightin 200 feet.” Realizing that driving often evokes a driver's memories oflandmarks such as signs, stores, and landscapes, some existingnavigation systems provide guidance by superimposing picture images upona fork or an intersection of a road depicting a landmark found thereinin a highlighted state.

SUMMARY

According to embodiments of the present invention, acomputer-implemented method for improving landmark-based route guidanceis provided. The computer-implemented method includes receiving, by aprocessor, image data including one or more landmarks. Thecomputer-implemented method further includes identifying, by theprocessor, a candidate landmark within the image data. Thecomputer-implemented method further includes presenting, by theprocessor, the candidate landmark to a user. The computer-implementedmethod further includes, in response to the user accepting the candidatelandmark, storing, by the processor, the candidate landmark and alocation of the candidate landmark.

According to embodiments of the present invention, a system forimproving landmark-based route guidance is provided. The system includesa memory having computer readable instructions and a processing devicefor executing the computer readable instructions. The computer readableinstructions cause the processing device to receive image data includingone or more landmarks and to identify a candidate landmark within theimage data. The computer readable instructions further cause theprocessing device to present the candidate landmark to a user and, inresponse to the user accepting the candidate landmark, store thecandidate landmark and a location of the candidate landmark.

According to embodiments of the present invention, a computer programproduct for improving landmark-based route guidance is provided. Thecomputer program product includes a computer readable storage mediumhaving program instructions embodied therewith, the program instructionsexecutable by a processing device to cause the processing device toperform a method. The method can include receiving image data includingone or more landmarks and identifying a candidate landmark within theimage data. The method can also include presenting the candidatelandmark to a user and, in response to the user accepting the candidatelandmark, storing the candidate landmark and a location of the candidatelandmark.

BRIEF DESCRIPTION OF THE DRAWINGS

The subject matter of the present invention is particularly pointed outand distinctly defined in the claims at the conclusion of thespecification. The foregoing and other features and advantages areapparent from the following detailed description taken in conjunctionwith the accompanying drawings in which:

FIG. 1 depicts a cloud computing environment according to an embodimentof the present invention;

FIG. 2 depicts abstraction model layers according to an embodiment ofthe present invention;

FIG. 3 depicts a block diagram of a processing system for improvinglandmark-based route guidance functionality according to one or moreembodiments of the present invention;

FIG. 4 depicts a block diagram of another processing system forimproving landmark-based route guidance functionality according to oneor more embodiments of the present invention;

FIG. 5 depicts a flow diagram of a method for implementing thepresentation and selection of candidate landmarks according to one ormore embodiments of the present invention;

FIG. 6 depicts a user view of a HUD and a display for implementing thepresentation of landmarks according to one or more embodiments of thepresent invention; and

FIG. 7 depicts a user view of additional HUDs for implementing thepresentation of landmarks according to one or more embodiments of thepresent invention.

DETAILED DESCRIPTION

In accordance with one or more embodiments of the invention, drivingassistant systems, methods, and computer program products for improvinglandmark-based route guidance functionality are provided. Variousembodiments of the present invention are described herein with referenceto the related drawings. Alternative embodiments can be devised withoutdeparting from the scope of this invention. References in thespecification to “one embodiment,” “an embodiment,” “an exampleembodiment,” etc., indicate that the embodiment described can include aparticular feature, structure, or characteristic, but every embodimentmay or may not include the particular feature, structure, orcharacteristic. Moreover, such phrases are not necessarily referring tothe same embodiment. Further, when a particular feature, structure, orcharacteristic is described in connection with an embodiment, it issubmitted that it is within the knowledge of one skilled in the art toaffect such feature, structure, or characteristic in connection withother embodiments whether or not explicitly described.

The following definitions and abbreviations are to be used for theinterpretation of the claims and the specification. As used herein, theterms “comprises,” “comprising,” “includes,” “including,” “has,”“having,” “contains” or “containing,” or any other variation thereof,are intended to cover a non-exclusive inclusion. For example, acomposition, a mixture, process, method, article, or apparatus thatcomprises a list of elements is not necessarily limited to only thoseelements but can include other elements not expressly listed or inherentto such composition, mixture, process, method, article, or apparatus.

Additionally, the term “exemplary” is used herein to mean “serving as anexample, instance or illustration.” Any embodiment or design describedherein as “exemplary” is not necessarily to be construed as preferred oradvantageous over other embodiments or designs. The terms “at least one”and “one or more” are understood to include any integer number greaterthan or equal to one, i.e. one, two, three, four, etc. The terms “aplurality” are understood to include any integer number greater than orequal to two, i.e. two, three, four, five, etc. The term “connection”can include an indirect “connection” and a direct “connection.”

For the sake of brevity, conventional techniques related to computerprocessing systems and abstraction models may or may not be described indetail herein. Moreover, it is understood that the various tasks andprocess steps described herein can be incorporated into a morecomprehensive procedure, process or system having additional steps orfunctionality not described in detail herein.

Turning now to a detailed description of the present invention, aspreviously noted herein, some existing navigation systems provideguidance by superimposing landmark images upon a fork or an intersectionof a road in a highlighted state. These landmarks are presented to thedriver to allow for a more natural route guidance experience.Conventional systems typically store landmark data client or server sidewith the bulk navigation data. The landmark data, which is otherwisestatic, can be periodically updated by the system.

Landmark-based navigation offers drivers several advantages over lesssophisticated systems. There are several problems, however, with theconventional approach to landmark-based navigation systems. First, thehighlighted landmark is not selected by the driver. Instead,conventional systems tag or otherwise associate landmark data withnavigation data and a particular landmark is automatically displayed tothe driver in response to the driver's location. As such, there is nonexus between what an individual driver recognizes as a landmark and thelandmark provided to that driver by the system. A driver can feelanxious when a landmark displayed by the system is different than thelandmark the driver remembers. Next, while the selection of landmarkscan be periodically updated, the landmarks themselves are not adaptive(i.e., the landmarks as displayed to a driver do not account for changesto the landmark itself over time). A driver can become confused when alandmark displayed by the system has been updated in response to achange in the landmark. For example, a driver that relies on aparticular Chinese noodle shop as a landmark can become confused whenthe system provides an image of a curry restaurant that recentlyreplaced the noodle shop. Consequently, the driver, looking for thenoodle shop, can make a wrong turn or fail to turn at the correctintersection. Thus, driving assistant systems, methods, and computerprogram products for improving landmark-based route guidancefunctionality are desired.

One or more embodiments provide a driving assistant system, acomputer-implemented method, and a computer product for associatinguser-specified landmarks with a route and for adaptively trackingchanges to the landmarks over time. The system continuously orperiodically receives image data from a camera. The system can identifyone or more candidate landmarks within the image data and can presentthe candidate landmarks to a user. The system stores the candidatelandmark and metadata associated with the candidate landmark in responseto the user confirming the candidate landmark for storage. In thismanner, the system is populated over time with a plurality of landmarksspecifically selected by the user. The selected landmarks can be moreeasily recognized by the user than stock or pre-generated landmarks. Thesystem can also compare a position of the user to the location of thelandmark. In response to determining that the position is within athreshold distance of the location, the system can compare the storedcandidate landmark to an image of the candidate landmark in the currentimage data to identify a change. This change can be presented to theuser. In this manner, the user is made aware of any changes in theappearance of the landmark which occurred after the user stored thelandmark.

One or more embodiments of the invention include or yield varioustechnical features, technical effects, and/or improvements totechnology. Example embodiments of the invention provide a drivingassistant system configured to perform an automatic, unsupervisedprocess to associate a user-identified landmark with a route location inresponse to receiving user input. These aspects of the inventionconstitute technical features that yield the technical effect of tagginga route with adaptive landmarks that avoids the need for manuallypopulating a database with landmark data and the technical effect ofusing a machine learning technique to progressively improve landmarkselection by identifying and storing preferred landmark types for futureuse. As a result of these technical features and technical effects, adriving assistant system in accordance with embodiments of the presentinvention represents an improvement to existing landmark-based routeguidance. It should be appreciated that the above examples of technicalfeatures, technical effects, and improvements to technology are merelyillustrative embodiments of the invention and are not exhaustive.

A driving assistant system, a computer-implemented method, and acomputer product for improving landmark-based route guidance inaccordance with one or more embodiments of the present invention aredescribed in detail below by referring to the accompanying drawings inFIGS. 1-7.

It is to be understood that although this disclosure includes a detaileddescription on cloud computing, implementation of the teachings recitedherein are not limited to a cloud computing environment. Rather,embodiments of the present invention are capable of being implemented inconjunction with any other type of computing environment now known orlater developed.

Cloud computing is a model of service delivery for enabling convenient,on-demand network access to a shared pool of configurable computingresources (e.g., networks, network bandwidth, servers, processing,memory, storage, applications, virtual machines, and services) that canbe rapidly provisioned and released with minimal management effort orinteraction with a provider of the service. This cloud model may includeat least five characteristics, at least three service models, and atleast four deployment models.

Characteristics are as follows:

On-demand self-service: a cloud consumer can unilaterally provisioncomputing capabilities, such as server time and network storage, asneeded automatically without requiring human interaction with theservice's provider.

Broad network access: capabilities are available over a network andaccessed through standard mechanisms that promote use by heterogeneousthin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

Resource pooling: the provider's computing resources are pooled to servemultiple consumers using a multi-tenant model, with different physicaland virtual resources dynamically assigned and reassigned according todemand. There is a sense of location independence in that the consumergenerally has no control or knowledge over the exact location of theprovided resources but can be able to specify location at a higher levelof abstraction (e.g., country, state, or datacenter).

Rapid elasticity: capabilities can be rapidly and elasticallyprovisioned, in some cases automatically, to quickly scale out andrapidly released to quickly scale in. To the consumer, the capabilitiesavailable for provisioning often appear to be unlimited and can bepurchased in any quantity at any time.

Measured service: cloud systems automatically control and optimizeresource use by leveraging a metering capability at some level ofabstraction appropriate to the type of service (e.g., storage,processing, bandwidth, and active user accounts). Resource usage can bemonitored, controlled, and reported providing transparency for both theprovider and consumer of the utilized service.

Service Models are as follows:

Software as a Service (SaaS): the capability provided to the consumer isto use the provider's applications running on a cloud infrastructure.The applications are accessible from various client devices through athin client interface such as a web browser (e.g., web-based e-mail).The consumer does not manage or control the underlying cloudinfrastructure including network, servers, operating systems, storage,or even individual application capabilities, with the possible exceptionof limited user-specific application configuration settings.

Platform as a Service (PaaS): the capability provided to the consumer isto deploy onto the cloud infrastructure consumer-created or acquiredapplications created using programming languages and tools supported bythe provider. The consumer does not manage or control the underlyingcloud infrastructure including networks, servers, operating systems, orstorage, but has control over the deployed applications and possiblyapplication hosting environment configurations.

Infrastructure as a Service (IaaS): the capability provided to theconsumer is to provision processing, storage, networks, and otherfundamental computing resources where the consumer is able to deploy andrun arbitrary software, which can include operating systems andapplications. The consumer does not manage or control the underlyingcloud infrastructure but has control over operating systems, storage,deployed applications, and possibly limited control of select networkingcomponents (e.g., host firewalls).

Deployment Models are as follows:

Private cloud: the cloud infrastructure is operated solely for anorganization. It can be managed by the organization or a third party andcan exist on-premises or off-premises.

Community cloud: the cloud infrastructure is shared by severalorganizations and supports a specific community that has shared concerns(e.g., mission, security requirements, policy, and complianceconsiderations). It can be managed by the organizations or a third partyand can exist on-premises or off-premises.

Public cloud: the cloud infrastructure is made available to the generalpublic or a large industry group and is owned by an organization sellingcloud services.

Hybrid cloud: the cloud infrastructure is a composition of two or moreclouds (private, community, or public) that remain unique entities butare bound together by standardized or proprietary technology thatenables data and application portability (e.g., cloud bursting forload-balancing between clouds).

A cloud computing environment is service oriented with a focus onstatelessness, low coupling, modularity, and semantic interoperability.At the heart of cloud computing is an infrastructure including a networkof interconnected nodes.

Referring now to FIG. 1, illustrative cloud computing environment 50 fordetermining whether a target compound can be synthesized using knownpathways for similar compounds is depicted. As shown, cloud computingenvironment 50 includes one or more cloud computing nodes 10 with whichlocal computing devices used by cloud consumers, such as, for example,personal digital assistant (PDA) or cellular telephone 54A, desktopcomputer 54B, laptop computer 54C, and/or automobile computer system 54Ncan communicate. Cloud computing nodes 10 can communicate with oneanother. They can be grouped (not shown) physically or virtually, in oneor more networks, such as Private, Community, Public, or Hybrid cloudsas described hereinabove, or a combination thereof. This allows cloudcomputing environment 50 to offer infrastructure, platforms and/orsoftware as services for which a cloud consumer does not need tomaintain resources on a local computing device. It is understood thatthe types of computing devices 54A-N shown in FIG. 1 are intended to beillustrative only and that computing nodes 10 and cloud computingenvironment 50 can communicate with any type of computerized device overany type of network and/or network addressable connection (e.g., using aweb browser).

Referring now to FIG. 2, a set of functional abstraction layers providedby cloud computing environment 50 (FIG. 1) is shown. It is understoodthat the components, layers, and functions shown in FIG. 2 are intendedto be illustrative only and that embodiments of the invention are notlimited thereto. As illustrated, the following layers and correspondingfunctions are provided:

Hardware and software layer 60 includes hardware and softwarecomponents. Examples of hardware components include: mainframes 61; RISC(Reduced Instruction Set Computer) architecture based servers 62;servers 63; blade servers 64; storage devices 65; and networks andnetworking components 66. In some embodiments, software componentsinclude network application server software 67 and database software 68.

Virtualization layer 70 provides an abstraction layer from which thefollowing examples of virtual entities can be provided: virtual servers71; virtual storage 72; virtual networks 73, including virtual privatenetworks; virtual applications and operating systems 74; and virtualclients 75.

In some embodiments, management layer 80 can provide the functionsdescribed below. Resource provisioning 81 provides dynamic procurementof computing resources and other resources that are utilized to performtasks within the cloud computing environment. Metering and Pricing 82provide cost tracking as resources are utilized within the cloudcomputing environment, and billing or invoicing for consumption of theseresources. In one example, these resources can include applicationsoftware licenses. Security provides identity verification for cloudconsumers and tasks, as well as protection for data and other resources.User portal 83 provides access to the cloud computing environment forconsumers and system administrators. Service level management 84provides cloud computing resource allocation and management such thatrequired service levels are met. Service Level Agreement (SLA) planningand fulfillment 85 provides pre-arrangement for, and procurement of,cloud computing resources for which a future requirement is anticipatedin accordance with an SLA.

Workloads layer 90 provides examples of functionality for which thecloud computing environment can be utilized. Examples of workloads andfunctions which can be provided from this layer include: mapping andnavigation 91; software development and lifecycle management 92; virtualclassroom education delivery 93; data analytics processing 94;transaction processing 95; and driving assistant container 96. In someembodiments, driving assistant container 96 provides all or part of thefunctionality of the processing system 400 (as depicted in FIG. 4).

FIG. 3 illustrates a block diagram of a processing system 100 forimproving landmark-based route guidance functionality according to oneor more embodiments. Processing system 100 can have one or more centralprocessing units (processors) 101 a, 101 b, 101 c, etc. (collectively orgenerically referred to as processor(s) 101 and/or as processingdevice(s) 101). In some embodiments, each processor 101 can include areduced instruction set computer (RISC) microprocessor. Processors 101are coupled to system memory (e.g., random access memory (RAM) 114) andvarious other components via a system bus 113. Read only memory (ROM)102 is coupled to system bus 113 and can include a basic input/outputsystem (BIOS), which controls certain basic functions of processingsystem 100.

Further illustrated are an input/output (I/O) adapter 107 and a networkadapter 106 coupled to system bus 113. I/O adapter 107 can be a smallcomputer system interface (SCSI) adapter that communicates with a harddisk 103, a tape unit 105, or any other similar component. I/O adapter107, hard disk 103, and tape unit 105 are collectively referred toherein as mass storage 104. Operating system 120 for execution onprocessing system 100 can be stored in mass storage 104. A networkadapter 106 interconnects system bus 113 with an outside network 116enabling processing system 100 to communicate with other such systems.

A display (e.g., a display monitor) 115 is connected to system bus 113by display adaptor 112, which can include a graphics adapter to improvethe performance of graphics intensive applications and a videocontroller. In some embodiments, adapters 106, 107, and/or 112 can beconnected to one or more I/O busses that are connected to system bus 113via an intermediate bus bridge (not shown). Suitable I/O buses forconnecting peripheral devices such as hard disk controllers, networkadapters, and graphics adapters typically include common protocols, suchas the Peripheral Component Interconnect (PCI). Additional input/outputdevices are shown as connected to system bus 113 via user interfaceadapter 108 and display adapter 112. A keyboard 109, mouse 110, andspeaker 111 can be interconnected to system bus 113 via user interfaceadapter 108, which can include, for example, a Super I/O chipintegrating multiple device adapters into a single integrated circuit.

In some embodiments, processing system 100 includes a graphicsprocessing unit 130. Graphics processing unit 130 is a specializedelectronic circuit designed to manipulate and alter memory to acceleratethe creation of images in a frame buffer intended for output to adisplay. In general, graphics processing unit 130 is very efficient atmanipulating computer graphics and image processing, and has a highlyparallel structure that makes it more effective than general-purposeCPUs for algorithms where processing of large blocks of data is done inparallel.

Thus, as configured herein, processing system 100 includes processingcapability in the form of processors 101, storage capability includingsystem memory (e.g., RAM 114), and mass storage 104, input means such askeyboard 109 and mouse 110, and output capability including speaker 111and display 115. In some aspects of the present invention, a portion ofsystem memory (e.g., RAM 114) and mass storage 104 collectively store anoperating system such as the AIX® operating system from IBM Corporationto coordinate the functions of the various components shown inprocessing system 100.

FIG. 4 illustrates a block diagram of a processing system 400 forimproving landmark-based route guidance functionality according to oneor more embodiments. The various components, modules, engines, etc.described regarding FIG. 4 can be implemented as instructions stored ona computer-readable storage medium, as hardware modules, asspecial-purpose hardware (e.g., application specific hardware,application specific integrated circuits (ASICs), as embeddedcontrollers, hardwired circuitry, etc.), or as some combination orcombinations of these. In some embodiments, the engine(s) describedherein can be a combination of hardware and programming. The programmingcan be processor executable instructions stored on a tangible memory,and the hardware can include processor 101 (FIG. 3) for executing thoseinstructions. Thus a system memory can store program instructions thatwhen executed by processor 101 implement the engines described herein.Other engines can also be utilized to include other features andfunctionality described in other examples herein.

Processing system 400 can include processor 101, camera 402, imagerecording module 404, image and landmark database 406, landmarkrecording module 408, third party data 410, user input 412, routemanagement module 414, driving data 416, map database 418, landmarkcomparison module 420, and presentation module 422. Image and landmarkdatabase 406 and map database 418 can each include a single database orone or more databases. In some embodiments, the image and landmarkdatabase 406 includes separate image databases and landmark databases.In some embodiments, the image and landmark database 406 and the mapdatabase 418 are a single database. In some embodiments, the image andlandmark database 406 can be stored on-vehicle or on a cloudcommunicatively accessible to the processing system 400 over a network.Alternatively or additionally, the processing system 400 can includededicated hardware, such as one or more integrated circuits, ApplicationSpecific Integrated Circuits (ASICs), Application Specific SpecialProcessors (ASSPs), Field Programmable Gate Arrays (FPGAs), or anycombination of the foregoing examples of dedicated hardware, forperforming the techniques described herein.

Image recording module 404 receives image data from the camera 402. Theimage data (also known as route images) can be recording continuously orperiodically. In some embodiments, a recording period setting ismanually selected by a user (e.g., a driver or a passenger) of theprocessing system 400. In some embodiments, the recording period settingis selected by the image recording module 404. In some embodiments, theimage recording module 404 starts recording image data when the driverstarts driving the vehicle (e.g., when the ignition is started or whenthe car moves). The image recording module 404 transfers image data tothe image and landmark database 406 and to the landmark comparisonmodule 420.

Landmark recording module 408 receives image data from the imagerecording module 404 as well as third party data 410 and user input 412.The landmark recording module 408 identifies landmarks within the imagedata received from the image recording module 404 and tags the landmarkswith route metadata in response to various triggers (i.e., data receivedfrom third party data 410 and/or user input 412). The route metadata caninclude, for example, GPS coordinates, a landmark name, a landmarklocation, shape, color, size, type, a time, weather at the time, arecording time, a season, a month, a year, a direction, whether thelandmark is a Q&A-type landmark, and/or any other suitable metadata foridentifying a particular landmark and matching the landmark to aparticular location. For example, route metadata for a copse of cherryblossom trees can include “Route 100 cherry blossoms”, GPS coordinates,March through April only, left the side of the road. In someembodiments, the metadata includes a user recording. The user recordingcan be provided, for example, by the user at the time of storing aregistered landmark. In some embodiments, the landmark recording module408 queries a user to provide any desired remarks associated with thelandmark. In this manner, the user can improve their ability to recallthe landmark in the future. For example, a user could record, “Thebuilding is on the left side under a large sign.” In some embodiments,the tagged landmarks are stored in the image and landmark database 406.The triggers can include, for example, contextual user speech triggers(e.g., Q&A-type landmarks), user command triggers, automatic routetriggers, third party triggers, and user selection triggers.

In some embodiments, the user input 412 includes voice data (speechdata) and the landmark recording module 408 includes a voice processingsection that monitors the vehicle for voice data containing contextualuser speech triggers. In some embodiments, the landmark recording module408 monitors the vehicle for voice data that indicates a “Q&A-typelandmark” which can be passively stored for future use by the systemwithout user confirmation. For example, a Q&A-type landmark speechtrigger including conversation data indicating that a user perceives alandmark, such as, for example, “Ah, it's over that sign, isn't it?”,“It's right after the statue”, “Look at those beautiful cherryblossoms”, or “You can see Mount Fuji” can cause the landmark recordingmodule 408 to stored the associated landmark (e.g., the sign, thestatue, the cherry blossoms, Mount Fuji) as a Q&A-type landmark at theuser's current location. Q&A-type landmarks are landmarks which are notautomatically presented to a user the next time the user returns to thelandmark location. Instead, Q&A-type landmarks are presented in responseto contextual voice triggers in a conversational manner. For example,the phrase “It is beyond the location where the cherry tree used to be.”can be presented to the user in response to detecting user or passengerconversation data including phrases such as “Is it near here?” at alocation near the previously recorded cherry tree. In another example, aQ&A-type landmark is passively stored by the landmark recording module408 in response to voice data including “Hey, take a look at the cherryblossoms. So beautiful!” The Q&A-type landmark metadata can include thelocation of the cherry blossoms, the season, and other metadatainformation according to one or more embodiments. Eight months laterwhen the user is traveling along the route having the cherry blossoms,the system detects conversation data including “This place looksdifferent than before, there is so much development now. Are we gettingclose to our turn?” The system can respond with “This is where thebeautiful cherry blossoms were last spring. Stay on the route until youreach the next intersection. Make a left.” In this manner, a Q&A-typelandmark can be presented to a user in a conversational manner to helpthe user remember the landmark and to boost the user's confidence thatthe correct route is being taken.

In some embodiments, speech triggers include user conversationsindicating a navigation error, such as, for example, “I took the wrongway” or “I should have made a turn there.” User commands can includeexplicit instructions to record a landmark, such as, for example,“Remember that sign”, “Save that blue building on the right” or “Save alandmark at the next intersection.” In some embodiments, the landmarkrecording module 408 identifies landmarks within current or recent imagedata in response to receiving the user commands.

In some embodiments, the landmark recording module 408 receivesautomatic route triggers from the route management module 414. Automaticroute triggers can include predetermined triggers, such as, for example,detour triggers. In some embodiments, the route management module 414can monitor the expected path that the vehicle is traveling and, inresponse to detecting that the vehicle has made a detour, the routemanagement module 414 can send an automatic route trigger to thelandmark recording module 408. In some embodiments, the route managementmodule 414 can send an automatic route trigger to the landmark recordingmodule 408 in response to detecting that the vehicle has returned to aroute after previously deviating from the route.

In some embodiments, the third party data 410 includes third partytriggers, i.e., landmark data generated by third party systems. Thethird party system data can include, for example, data from otherdrivers using improved landmark-based driving assistant systemsaccording to one or more embodiments. In some embodiments, the landmarkrecording module 408 filters the received third party triggers using aquality threshold. For example, the landmark recording module 408 canrequire that a particular third party trigger be used by a predeterminednumber or percentage of users prior to relying on the third partytrigger. In this manner, only the most useful third party triggersresult in the landmark recording module 408 identifying landmark for theuser. The quality threshold can be predetermined by the processingsystem 400 or manually configured by the user. In some embodiments, theuser can increase or decrease the quality threshold to adjust the numberof third party landmarks utilized by the landmark recording module 408.

In some embodiments, the landmark recording module 408 carries out theprocess of recording a landmark within an image and its associatedmetadata in response to an intentional operation made by the user (i.e.,a user selection trigger). The user input 412 can include, for example,user interface menu selections made by the user to initiate a landmarkselection.

Once the landmark recording module 408 receives a trigger, the landmarkrecording module 408 identifies landmarks within the image data receivedfrom the image recording module 404 and tags the landmarks with routemetadata (i.e., the landmark is registered). In some embodiments, thelandmark recording module 408 uses known image recognition techniques toidentify candidate landmarks within the image data. In some embodiments,the landmark recording module 408 presents the candidate landmarks tothe user for confirmation. In some embodiments, the landmark recordingmodule 408 elicits voice-based confirmations from the user during thecandidate landmark approval process. For example, the landmark recordingmodule 408 can ask the user, “It is this yellow building on the right,isn't it?.”

The user can accept or reject the candidate landmark. If accepted, thelandmark is tagged with route metadata and stored in the image andlandmark database 406. If rejected, the landmark recording module 408can present a different candidate landmark to the user for confirmation.In this manner, a user can cycle through the candidate landmarks toselect a landmark best suited to the individual user. In someembodiments, the user can suspend the approval of a candidate landmarkfor review at a later time. In some embodiments, the landmark recordingmodule 408 saves the time and route position at which the suspension wasdesignated to ease later review. In some embodiments, the landmarkrecording module 408 presents candidate route metadata for a landmarkafter user approval. In some embodiments, the user can approve, reject,or modify the metadata. The landmark recording module 408 stores thetagged landmark after receiving approval of the landmark and metadata.In some embodiments, the user can cancel the landmark registration atany step of the landmark registration process.

In some embodiments, additional tools are provided to increase theaccuracy (i.e., the likelihood that a candidate landmark is ultimatelyaccepted by the user) of the candidate landmarks presented to the userby the landmark recording module 408. For example, the landmarkrecording module 408 can include a line of sight detector to monitor, inreal-time, the line of sight data of a user. In response to receiving atrigger, the landmark recording module 408 can use the line of sightdata to more accurately predict a candidate landmark. For example, whenan utterance “that sign” is made, a candidate of a sign that serves as alandmark is selected from target objects near the direction of the lineof sight of the driver and the candidate landmark is presented to thedriver. In some embodiments, the tendency of a user to select landmarksof a particular type or of a particular quality (e.g., route metadata)can be learned using machine learning techniques and a candidatelandmark of that type can be preferentially presented. In this manner,the landmark recording module 408 can adapt to particular users toimprove the selection of personalized landmarks. For example, signs arepreferentially presented as candidate landmarks to a driver whofrequently selects signs as route landmarks, and buildings havingparticular colors, hue, and brightness characteristics arepreferentially presented to a driver who frequently selects buildingshaving those particular colors, hue, and brightness as landmarks.

In some embodiments, a user (e.g., a driver or a passenger) can enteruser input 412 into the route management module 414 to begin routeguidance. The user input 412 includes route selection settings, such as,for example, a destination address. The route management module 414provides route guidance using known navigational techniques, based on,for example, driving data 416 and a map database 418. The driving data416 can include, for example, the current GPS coordinates of thevehicle. In some embodiments, the route management module 414 presentsroute guidance through the presentation module 422. In some embodiments,the presentation module 422 includes a screen and route guidance isdisplayed to the user on the screen. In some embodiments, thepresentation module 422 is coupled to the vehicles sound or speakersystem and route guidance is provided to the user through the speakers.In some embodiments, the presentation module 422 uses a combination ofvisual and verbal instructions to guide the user. In some embodiments,the presentation module 422 includes a windshield projection (also knownas a heads-up display, or HUD) for displaying images. In someembodiments, the presentation module 422 can project an image of alandmark and/or a dotted line or other demarcation onto the windshieldin the relative direction of the real-world landmark.

In some embodiments, the processing system 400 includes a landmarkcomparison module 420. The landmark comparison module 420 monitors routedata received from the route management module 414 to determine when thevehicle is approaching a registered landmark. When the vehicleapproaches a location corresponding to a landmark stored in the imageand landmark database 406, the landmark comparison module 420 comparesthe image of the landmark as stored in the image and landmark database406 against the visual depiction of the landmark within the image datacaptured by the camera 402.

In the case where no change is detected between the recorded image ofthe landmark and the visual depiction of the landmark, the landmarkcomparison module 420 presents the landmark to the user via thepresentation module 422 according to one or more embodiments.

In the case where a change is detected (e.g., a color of a building, acategory of a business, a content of a sign, etc.) between the recordedimage of the landmark and the visual depiction of the landmark, thelandmark comparison module 420 provides a guidance regarding thedifference to the user via the presentation module 422. In someembodiments, the guidance is in the form of an image, voiceinstructions, or a combination of visual and audio cues. For example,the guidance can include, “The yellow building is now painted blue.”,“There is a Mexican restaurant in place of the movie theater.”, “Thesign of the buckwheat noodle shop has been replaced by a sign of ahotel. The color of the sign is white.” or “The forest has beendeveloped into a residential area.” The guidance can also include, forexample, a pair of HUD images depicting the change in the landmark. Insome embodiments, the landmark comparison module 420 prompts the user tostore an updated image of the landmark. In some embodiments, thelandmark comparison module 420 automatically stores an updated image ofthe landmark.

In the case where the landmark no longer exists, the landmark comparisonmodule 420 provides a guidance informing the user via the presentationmodule 422. For example, the guidance can include, “The red fast foodrestaurant which used to be 100 feet ahead on the left side no longerexists.” In some embodiments, the landmark comparison module 420provides alternative landmarks, such as, for example, “Look instead fora firehouse on the right.” In some embodiments, the landmark comparisonmodule 420 provides guidance by means of an image and a voice describingthe fact that the place the automobile itself is traveling is where thelandmark used to be with a description of the difference between thepast (e.g., the expected landmark) and the current state (e.g., analternative landmark). By the explicit presentation of the fact that thelocation at issue is where the landmark existed, it is made possible toclearly present to the driver the fact that the driver is not drivingalong an erroneous route. In some embodiments, the landmark comparisonmodule 420 can prompt the user to approve a new landmark to replace thelandmark that no longer exists.

In the case where the visibility of the landmark has changed due, forexample, to the time of day, weather, or construction, the landmarkcomparison module 420 provides a guidance informing the user that thelandmark exists but that the visibility of the landmark is impaired. Insome embodiments, the guidance includes a “current visibility”description. The “current visibility” can be based on either the imagedata received from the camera 402 or the image data within the thirdparty data 410.

In some embodiments, the landmark as previously recorded by the user iscompared against the landmark as obtained by relatively recent thirdparty data 410 (i.e., third party data 410 recorded after the landmarkrecorded by the user) according to one or more embodiments. In thismanner it is possible to compare the stored images of the landmarkbefore the landmark can be seen from the vehicle itself and theinformation can be presented to the user ahead of time. For example, “Inone mile look for a two-story building on the left. The last time yousaw the building it was white. Others have reported that the building isnow red.” Consequently, the user can be informed of any changes to alandmark so that the user can look for the landmark as it currentlyexists.

In some embodiments, the presentation module 422 alters the guidancebased on a positional relationship between a landmark location (e.g.,the actual location of the landmark) and a route location associatedwith the landmark (e.g., an intersection or fork at which a vehiclemakes a turn). When the distance between the landmark location and theroute location is small (e.g., less than 2000 ft., 1000 ft., 500 ft.,100 ft., 50 ft.) the guidance is provided at the point that the vehiclepasses the route location. For example, “Turn right at the trafficlight. You will see a red and black Chinese noodle shop to the left ofthe traffic light.” When the distance between the landmark location andthe route location is large (e.g., more than 2000 ft., 1000 ft., 500ft., 100 ft., 50 ft.) or when the landmark is a wide-area landscape(e.g., a “residential area” or a mountain in the distance) the guidanceis provided before the vehicle reaches the route location. For example,“In one mile you should be able to see a castle on top of the hill tothe left.” In some embodiments, distance and time estimates included inthe guidance are calculated based on past driving history (e.g., speedor average user and/or third party time from current location to theroute location). For example, “When you see the castle on the top of thehill to the left, you will be approximately five minutes from a fork.Take the right side.”

In some embodiments, one or more additional landmarks are availablebetween a registered landmark and the user's current location. Thepresentation module 422 can supplement the guidance with the additionallandmarks. For example, “After you pass a statue on the right, gostraight along a long downhill and turn right at the third trafficlight.”

FIG. 5 illustrates a flow diagram of a method 500 for implementing thepresentation and selection of candidate landmarks according to one ormore embodiments. As shown at block 502, a trigger (here, a speechtrigger uttered by a user, e.g., “Oh, I should've made a turn there.”)is detected by the landmark recording module 408 according to one ormore embodiments.

As shown at block 504, the landmark recording module 408, detecting thatthe speech trigger includes user conversations indicating a navigationerror, asks the user if the user would like to record a landmark toavoid similar navigational issues in the future (e.g., “Would you liketo record a landmark for this location?”).

As shown at block 506, the user approves the presentation of candidatelandmarks (e.g., “Yes, present landmarks.”). In response, at block 508the landmark recording module 408 presents the first candidate landmarkon a display of the presentation module 422 according to one or moreembodiments.

As shown at block 510, the user rejects the first candidate landmark(e.g., “That's not so visible. Are there other candidates?”). Inresponse, at block 512 the landmark recording module 408 presents asecond candidate landmark on the display (e.g., “How about this sign?”).

As shown at block 514, the user rejects the second candidate landmark(e.g., “Too small. Give me another one.”). In response, at block 516 thelandmark recording module 408 presents a third candidate landmark on thedisplay (e.g., “How about this building?”).

As shown at block 518, the user accepts the third candidate landmark(e.g., “Ah, I remember this. I'll take it.”). In response, at block 520the landmark recording module 408 presents candidate metadata to theuser (“Yellow, two-story building.”).

As shown at block 522 the user accepts the candidate metadata (“Yes,that is correct.”). In response, at block 524 the landmark recordingmodule 408 tags the third candidate landmark with the accepted routemetadata, stores the landmark in the image and landmark database 406,and informs the user (“The yellow, two-story building has been saved asa landmark.”) according to one or more embodiments.

Additional processes also can be included, and it should be understoodthat the processes depicted in FIG. 5 represent illustrations, and thatother processes can be added or existing processes can be removed,modified, or rearranged without departing from the scope and spirit ofthe present invention. For example, the user could request additionalmetadata, or entirely different metadata, according to one or moreembodiments. In another example, the user could suspend the process atany time for later completion, according to one or more embodiments.

FIG. 6 illustrates a user view of a HUD 602 and a display 604 forimplementing the presentation of landmarks according to one or moreembodiments. The HUD 602 includes a visual guidance 606 (here, abroken-line frame) framing a landmark 608. A previously registered image610 of the landmark 608 is provided adjacent to the visual guidance 606.In this manner, the user can quickly compare the visual guidance 606 tothe registered image 610 of the landmark 608. The HUD 602 also includesa turn signal 612. The display 604 includes a user interface 614 (e.g.,Map, Info, Setting, and Menu buttons) and a map display 616 having aturn signal 618 according to one or more embodiments. The display 604also includes a visual guidance 620 and a comparison image 622. In someembodiments, the visual guidance 620 depicts a zoomed in view of thelandmark 608 as captured by the camera 402 in real-time.

FIG. 7 illustrates a user view of a HUD 702 and a HUD 704 forimplementing the presentation of landmarks according to one or moreembodiments. The HUD 702 includes a visual guidance 706 framing adifficult to see landmark 708 (perhaps, e.g., the landmark 708 is notobstructed but is difficult to see at night). A previously registeredimage 710 depicting the landmark 708 under daylight conditions isprovided adjacent to the visual guidance 706. In this manner, the useris able to view an image of the landmark 708 under better visibilityconditions. The HUD 704 includes a visual guidance 712 depicting ananticipated registered landmark 714 a kilometer ahead on the driver'sroute. A comparison image 716 showing that the landmark 714 has changedcolor is provided adjacent to the visual guidance 712. In this manner,the user is able to anticipate the visual change in the landmark 714 andmore easily spot the landmark 714 along the route.

The present techniques can be implemented as a system, a method, and/ora computer program product. The computer program product can include acomputer readable storage medium (or media) having computer readableprogram instructions thereon for causing a processor to carry outaspects of the present invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium can be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network can includecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention can be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions can execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer can be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection can be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) can execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitryand to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to aspects of thepresent invention. It is understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions can be provided to aprocessor of a special purpose computer or other programmable dataprocessing apparatus to produce a machine, such that the instructions,which execute via the processor of the computer or other programmabledata processing apparatus, create means for implementing thefunctions/acts specified in the flowchart and/or block diagram block orblocks. These computer readable program instructions can also be storedin a computer readable storage medium that can direct a computer, aprogrammable data processing apparatus, and/or other devices to functionin a particular manner, such that the computer readable storage mediumhaving instructions stored therein includes an article of manufactureincluding instructions which implement aspects of the function/actspecified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions can also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousaspects of the present invention. In this regard, each block in theflowchart or block diagrams can represent a module, segment, or portionof instructions, which includes one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block can occur out of theorder noted in the figures. For example, two blocks shown in successioncan, in fact, be executed substantially concurrently, or the blocks cansometimes be executed in the reverse order, depending upon thefunctionality involved. It is understood that each block of the blockdiagrams and/or flowchart illustration, and combinations of blocks inthe block diagrams and/or flowchart illustration, can be implemented byspecial purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

The descriptions of the various embodiments of the present inventionhave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments described. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the invention.The terminology used herein was chosen to best explain the principles ofthe embodiment, the practical application or technical improvement overtechnologies found in the marketplace, or to enable others of ordinaryskill in the art to understand the embodiments described herein.

What is claimed is:
 1. A computer-implemented method comprising:receiving, by a processor, image data comprising one or more landmarks;identifying, by the processor, a candidate landmark within the imagedata; presenting, by the processor, the candidate landmark to a user;and in response to the user accepting the candidate landmark, storing,by the processor, the candidate landmark and a location of the candidatelandmark.
 2. The computer-implemented method of claim 1, furthercomprising: comparing, by the processor, a position of the user to thelocation; in response to determining that the position is within athreshold distance of the location, comparing, by the processor, thestored candidate landmark to an image of the candidate landmark in theimage data to identify a change; and presenting, by the processor, thechange to the user.
 3. The computer-implemented method of claim 1,further comprising: in response to the user rejecting the candidatelandmark, presenting, by the processor, a new candidate landmark to theuser.
 4. The computer-implemented method of claim 1, further comprising:in response to the user accepting the candidate landmark, tagging, bythe processor, the candidate landmark with landmark metadata.
 5. Thecomputer-implemented method of claim 4, wherein the metadata comprisesGPS coordinates, a landmark name, a landmark location, a shape, a color,a size, a type, a time, weather at the time, a season, a month, a year,a direction, a relative position, or a user recording.
 6. Thecomputer-implemented method of claim 1, wherein receiving the image datafurther comprises receiving, by the processor, the image data from avehicle-mounted camera.
 7. The computer-implemented method of claim 1,wherein identifying a candidate landmark within the image data is inresponse to receiving a trigger.
 8. The computer-implemented method ofclaim 7, wherein the trigger comprises contextual user speech triggers,user command triggers, automatic route triggers, third party triggers,or user selection triggers.
 9. The computer-implemented method of claim1, wherein identifying a candidate landmark within the image datafurther comprises detecting a line of sight of the user.